Eliciting Forecasts from Self-interested Experts: Scoring 

Rules for Decision Makers 



Craig Boutilier 

Department of Computer Science 
University of Toronto, Canada 
cebly@cs.toronto.edu 



ABSTRACT 

Scoring rules for eliciting expert predictions of random variables are usu- 
ally developed assuming that experts derive utility only from the quality of 
their predictions (e.g., score awarded by the rule, or payoff in a prediction 
market). We study a more realistic setting in which (a) the principal is a 
decision maker and will take a decision based on the expert's prediction; 
and (b) the expert has an inherent interest in the decision. For example, in 
a corporate decision market, the expert may derive different levels of utility 
from the actions taken by her manager. As a consequence the expert will 
usually have an incentive to misreport her forecast to influence the choice of 
the decision maker if typical scoring rules are used. We develop a general 
model for this setting and introduce the concept of a compensation rule. 
When combined with the expert's inherent utility for decisions, a compen- 
sation rule induces a net scoring rule that behaves like a normal scoring 
rule. Assuming full knowledge of expert utility, we provide a complete 
characterization of all (strictly) proper compensation rules. We then ana- 
lyze the situation where the expert's utility function is not fully known to 
the decision maker. We show bounds on: (a) expert incentive to misreport; 
(b) the degree to which an expert will misreport; and (c) decision maker loss 
in utility due to such uncertainty. These bounds depend in natural ways on 
the degree of uncertainty, the local degree of convexity of net scoring func- 
tion, and natural properties of the decision maker's utility function. They 
also suggest optimization procedures for the design of compensation rules. 
Finally, we briefly discuss the use of compensation rules as market scoring 
rules for self-interested experts in a prediction market. 

1. INTRODUCTION 

Eliciting predictions of uncertain events from experts or other 
knowledgeable agents — or relevant information pertaining to events — 
is a fundamental problem of study in statistics, economics, opera- 
tions research, artificial intelligence and a variety of other areas 
|16| [5). Increasingly, robust mechanisms for prediction are being 
developed, proposed and/or applied in real-world domains rang- 
ing from elections and sporting events, to events of public inter- 
est (e.g., disease spread or terrorist action), to corporate decision 
making. Indeed, the very idea of crowd-sourcing and information 
(or prediction) markets is predicated on the existence of practical 
mechanisms for information elicitation and aggregation. 

A key element in any prediction mechanism involves providing 
an expert agent with the appropriate incentives to reveal a forecast 
they believe to be accurate. Many forms of "outcome-based" scor- 
ing rules, either individual or market-based, provide experts with 
incentives to (a) provide sincere forecasts; (b) invest effort to im- 
prove the accuracy of their personal forecasts; and (c) participate 



in the mechanism if they believe they can improve the quality of 
the principal's forecast. However, with just a few exceptions (see, 
e.g., |15||13| [3][7), most work fails to account for the ultimate use 
to which the forecast will be put. Furthermore, even these models 
assume that the experts who provide their forecasts derive no util- 
ity from the final forecast, or how it will be used, except insofar as 
they will be rewarded by the prediction mechanism itself. 

In many real-world uses of prediction mechanisms, this assump- 
tion is patently false. Setting aside purely informational and en- 
tertainment uses of information markets, the principal is often in- 
terested in exploiting the elicited forecast in order to make a de- 
cision |10[ 1 1 3 1 13). In corporate prediction markets, the principal 
may base strategic business decisions on internal predictions of un- 
certain events. In a hiring committee, the estimated probability of 
various candidates accepting offers (and being given offers by com- 
petitors) will influence the order in which (and whether) offers are 
made. Of course, other examples abound. Providing appropriate 
incentives in the form of scoring rules is often difficult in such set- 
tings, especially when the outcome distribution is conditional on 
the decision ultimately taken by the principal |13| |3). However, 
just as critically, in these settings, the experts whose forecasts are 
sought often have their own interests in seeing specific decisions 
being taken, interests that are not (fully) aligned with those of the 
principal. For example, in a corporate setting, an expert from a 
certain division may have an incentive to misreport demand for 
specific products, thus influencing R&D decisions that favor her 
division. In a hiring committee, an committee member may misre- 
port the odds that a candidate will accept a competing position in 
order to bias the "offer strategy" in a way that favors his preferred 
candidate. 

In this work, we develop what we believe to be the first class 
of scoring rules that incentivizes truthful forecasts even when ex- 
perts have an interest in the decisions taken by the principal, and 
hence would like to provide forecasts that manipulate that decision 
"in their favor." Other work has studied both decision making and 
incentive issues in prediction markets, but none that we are aware 
of addresses the natural question of expert self-interest in the deci- 
sions of the principal. 

Hanson 1 10 1 introduced the term decision markets to refer to the 
broad notion of prediction markets where experts offer forecasts for 
events conditional on some policy being adopted or a decision be- 
ing taken. Othman and Sandholm 1 13 J provide the first explicit, for- 
mal treatment of a principal who makes decisions based on expert 
forecasts. They address several key difficulties that arise due to the 
conditional nature of forecasts, but assume that the experts them- 
selves have no direct interest in the decision that is taken. Chen 
and Kash 1 3 1 extend this model to a wider class of informational 
settings and decision policies. Dimitrov and Sami 1 7 1 consider the 
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issue of strategic behavior across multiple markets and the possi- 
bility that an expert may misreport her beliefs in one market to ma- 
nipulate prices (and hence gain some advantage) in another. Simi- 
larly, Conitzer |6] explores strategic aspects of prediction markets 
through their connections to mechanism design. While the mech- 
anism design approach could prove very useful for the problems 
we address (see concluding remarks in Sec. [6J, Conitzer assumes 
an expert's utility is derived solely from the payoff provided by the 
prediction mechanism. Also related to the model we develop here 
is the analysis of Shi et al. 1 15 1, who consider experts that, once 
they report their forecasts, can take action to alter the probabilities 
of the outcomes in question. Unlike our model, they do not con- 
sider expert utility apart from the payoff offered by the mechanism 
(though, as in our model, the principal does have a utility function 
that dictates the value of an expert report). 

Our basic building block is a scoring rule for a single expert 
who knows the principal's policy — i.e., mapping from forecasts to 
decisions — and where the principal knows the expert's utility for 
decisions. We show that the scoring rule must compensate the ex- 
pert in a very simple and intuitive way based on her utility function. 
Specifically, the principal uses a compensation function that, when 
added to the inherent utility the expert derives from the principal's 
decision, induces a proper scoring rule. In a finite decision space, 
an expert's optimal utility function is piecewise-linear and convex 
in probability space — we describe one natural scoring rule based 
on this function that is proper, but not strictly so. We then provide 
a complete characterization of all proper compensation functions. 
We also characterize those which, in addition, satisfy weak and 
strong participation constraints that ensure an expert will be suffi- 
ciently compensated to "play the game." 

We then provide a detailed analysis of both expert uncertainty in 
the principal's policy, and principal uncertainty in the expert's util- 
ity for decisions. First we observe that the expert need not know the 
principal's policy prior to providing her forecast as long as she can 
verify the decision taken after the fact. Second, we analyze the im- 
pact of principal uncertainty regarding the expert's utility function. 
In general, the principal cannot ensure truthful reporting. However, 
we show that, given bounds on this uncertainty, bounds can then be 
derived on all of the following: (i) the expert's incentive to misre- 
port; (ii) the deviation of the expert's misreported forecast from its 
true beliefs; and (iii) the loss in utility the principal will realize due 
to this uncertainty. The first two bounds rely on the notion of strong 
convexity of the net scoring function induced by the compensation 
rule. The third bound uses natural properties of the principal's util- 
ity function. Apart from bounds derived from global strong con- 
vexity, we show that these bounds can be significantly tightened 
using local strong convexity, specifically, by ensuring merely that 
the net scoring function is sufficiently (and differentially) strongly 
convex near the decision boundaries of the principal's policy. These 
bounds suggest computational optimization methods for for design- 
ing compensation rules (e.g., using splines related to the principal's 
utility function). We conclude by briefly discussing a market scor- 
ing rule (MSR) based on our one-shot compensation rule. Using 
this MSR, the principal may need to provide more generous com- 
pensation to each expert than in the one-shot case, simply to ensure 
participation; but in some special cases, no additional compensa- 
tion is needed. 

The paper is organized as follows. We begin with a basic back- 
ground on scoring rules for prediction mechanisms in Sec. [2] In 
Sec. [3] we define our model for analyzing the behavior of self- 
interested experts, introduce compensation rules, and show that the 
resulting net scoring function can be used to analyze expert behav- 
ior. We provide a complete characterization of (strict) proper com- 



pensation rules and and further characterize two subclasses of com- 
pensation rules that satisfy the two participation constraints men- 
tioned above. In Sec.|4]we relax two assumptions in our model. We 
first show the expert need to be aware of the principal's policy for 
our model to work. We then consider a principal that has imperfect 
knowledge of the expert's utility function, and using the notion of 
(local and global) strong convexity derives bounds on the expert's 
incentive to misreport and the impact on the quality of the princi- 
pal's decision. After a brief discussion of market scoring rules in 
Sec. [5] we conclude in Sec.[6]wifh a discussion of several avenues 
for future research. 

2. BACKGROUND: SCORING RULES 

We begin with a very brief review of relevant concepts from the 
literature on scoring rules and prediction markets. For comprehen- 
sive overviews, see the surveys |16|[5). 

We assume that an agent — the principal — is interested in assess- 
ing the distribution of some discrete random variable X with fi- 
nite domain X = {xi, . . . ,x m }. Let A(X) denote the set of 
distributions over X, where p £ A(X) is a nonnegative vector 
(pi, . . . ,p m ) s.t. Y2iP* — 1- The principle can engage one or 
more experts to provide a forecast p £ A(X). We focus first on 
the case of a single expert E. For instance, to consider a simple 
toy example we use throughout the sequel, the mayor of a small 
town may ask the local weather forecaster to offer a probabilistic 
estimate of weather conditions for the following weekend^ 

We assume E has beliefs p about X, but a key question is how to 
incentivize E to report p faithfully (and devote reasonable effort to 
developing accurate beliefs). A variety of scoring rules have been 
developed for just this purpose |2 14] [12] [8). A scoring rule is a 
function S : A(X) x X — > R that provides a score (or payoff) 
S(r, Xi) to E if she reports forecast r and the realized outcome of 
X is Xi, essentially rewarding E for her predictive "performance" 
(2] [12) . If E has beliefs p and reports r, her expected score is 
S(r, p) = Y2i , S'( r ! x i)Pi- We say S is a proper scoring rule iff a 
truthful report is optimal for E: 

S(p,p)>5(r,p), Vp,reA(X) (1) 

We say that S is strictly proper if inequality {T} is strict for r 7^ p 
(i.e., E has strict disincentive to misreport). A variety of strictly 
proper scoring rules have been developed, among the more popular 
being the log scoring rule, where S(p,Xi) = alogp^ + hi (for 
arbitrary constants a > and bi) 1 1 2[| I4| . In what follows, we will 
restrict attention to regular scoring rules in which payment S(r, Xi) 
is bounded whenever r% > 0. 

Proper scoring rules can be fully characterized in terms of convex 
cost function s |12||14| ; here we review the formulation of Gneiting 
and Raftery (8pLet G : A(X) -> R be any convex function 
over distributions — we refer to G as a cost function. We denote 
by G* : A(X) — > R m some subgradient of G, i.e., a function 
satisfying 

G(q) > G(p)+G*(p)-(q-p) 

for all p, q g A(X)F] Such cost functions and associated subgra- 
dients can be used to derive any proper scoring rule. 

THEOREM 1. ^72] [74] A regular scoring rule S is proper iff 

S(p, Xi) = G(p) - G*(p) ■ (p) + G* (p) (2) 

'More significant examples in the domains of public policy or cor- 
porate decision making, as discussed above, can easily be con- 
structed by the reader. 

2 If G is differentiable at p then the subgradient at that point is 
unique, namely, the gradient VG(p). 
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Figure 1: Illustration of a proper scoring rule. If the expert 
reports p, its expected score (relative to its true beliefs) is given 
by the subtangent hyperplane H p . For any report q different 
from p, the expected score S(q,p) = H q ■ p must be less than 
the expected score S(p,p) = H p ■ p of truthful reporting. 

for some convex G and subgradient G* . S is strictly proper iff G 
is strictly convex. 

Intuitively, Eq.[2]defines a hyperplane 

h p = {S{p,xi),...,S(p,x m )} y 

for each point p, that is subtangent to G at p. This defines a linear 
function, for any fixed report p, giving the expected score of that 
report given beliefs q: S(p, q) = H p ■ q. An illustration is given 
in Fig.[T|for a simple one-dimensional (two-outcome) scenario. 

There are a number of prediction market mechanisms that allow 
the principal to extract information from multiple experts; see |16| 
5] for excellent surveys. Here we focus on market scoring rules 
(MSRs) (9] |11| , which allow experts to (sequentially) change the 
forecasted p using any proper scoring rule S. Given the current 
forecast p', an expert can change the forecast to p if she is willing 
to pay according to S(p' , •) and receive payment S(p, ■). If her 
true beliefs p are different from p' and the scoring rule is strictly 
proper, then she has incentive to participate and report truthfully. 
Under certain conditions, MSRs can be interpreted as automated 
market makers (4). Since each expert pays the amount due to the 
previous expert for her prediction, the net payment of the principal 
is the score associated with the final prediction. 

3. SCORING RULES FOR SELF-INTERESTED 
EXPERTS 

Scoring rules in standard models assume that an expert offering 
a forecast is uninterested in any aspect her forecast other than the 
score she will derive from her prediction. As discussed above, there 
are many settings where the principal will make a decision based 
on the received forecast, and the expert has a direct interest in this 
decision. In this section, we develop a model for this situation and 
devise a class of scoring rules that incentive self-interested agents 
to report their true beliefs. 

3.1 Model Formulation 

We assume the principal, or decision maker (DM), will elicit a 
forecast of X from expert E, and make a decision that is influenced 
by this forecast. Let D — {di, . . . , d n } be the set of possible de- 
cisions, and Uij be DM's utility should it take decision di with Xj 
being the realization of X. Letting u^ = (itji, • • • , u im ), the ex- 
pected utility of decision di given distribution p is Ui (p) = Uj • p. 
For any beliefs p, DM will want to take the decision that max- 
imizes expected utility, giving DM the utility function U (p) = 
maxi<„ Ui(p). Since each Ui is a linear function of p, U is piece- 
wise linear and convex (PWLC). Furthermore, each di is optimal 



in a (possibly empty) convex region of belief space Di = {p : 
Ui • p > Uj • p, Vj}. We assume DM acts optimally and that it has 
a policy n : A (A) — s> D that selects some optimal decision 7r(p) 
for any expert forecast p. In what follows, we take Di — 7v~ (di). 
We denote by Dij the (possibly empty) boundary between Di and 
Dj. Notice that for any p € Dij we must have Ut(p) = Uj (p)^\ 

In our running example, suppose the mayor must decide whether 
to hold a civic ceremony in the town park or at a private banquet fa- 
cility. Given a forecast probability p of rain, she will make outdoor 
arrangements at the park if p falls below some threshold r, and will 
rent the banquet facility if p is above r (here r is the indifference 
probability: U p . drk (r) = (TbanqO))- 

We note that this model of DM utility is slightly more restricted 
than that of [ 13 3 1, who allow the utility of each decision to depend 
on a different random variable, and assume that the realization of 
a variable will be observed only if the corresponding decision is 
taken. This introduces difficulties in offering suitable incentives for 
participation that do not arise in our setting; indeed, the primary 
contribution of Othman and Sandholm |6|, and the extension by 
Chen and Kash 1 3 1, is a characterization of a form of proper scoring 
in the face of these complications. We also confine our attention 
primarily to a principal that maximizes its expected utility given 
_B's report (in the terminology of [13], DM uses the max decision 
rule), though we remark on the possible use of stochastic policies 
by DM in Sec. [4^] 

Now suppose a single expert E is asked to provide a forecast 
of X that permits DM to make a decision. Assume that E knows 
DM's policy n: knowledge of DM's utility function is sufficient for 
knowledge of the policy but is not required (we discuss the possi- 
bility of E being uncertain about n in Sec. |4. Further, assume 
that E has its own utility function or bias b, where bij is E's utility 
should DM take decision di and Xj is the realization of X. Define 
bi = (6ii, • ■ • , bi m ); and let E's expected utility for di given p be 
Bi (p) = bj • p. In our small example, the weather forecaster may 
be related to the owner of the banquet facility, and will get some de- 
gree of satisfaction (or a small kickback) if the mayor's ceremony 
is held there. 

As with DM, E's optimal utility function B* (if DM were acting 
on E's behalf) is PWLC: 

B*(p) — maxb, • p. (3) 

i 

Denote by D*(p) the decision di that maximizes Eq.[3] i.e., E's 
preferred decision given beliefs p (see Fig.|2]for an illustration). 

Of course, DM is pursuing its own policy n, not acting to op- 
timize E's utility. Hence £"s actual utility for a specific report r 
under beliefs p is given by 

B^{r,p) = b^ (r) ■ p; (4) 

that is, if she reports r, DM will take decision 7r(r) = d^ for some 
k, and she will derive benefit Bk(p). We refer to B n (r, p) as E's 
inherent utility for reporting r. Similarly, B(r,Xi) — bi^(r) is 
E's inherent utility for report r under realization Xi. This is simply 
the inherent benefit she derives from the decision she induces DM 
to take. This is illustrated in Fig. [2] Notice that E's utility for 
reports, given any fixed beliefs p, is not generally continuous, with 
potential (jump) discontinuities at DM's decision boundaries. 

Without some scoring rule, there is a clear incentive for E to 
misreport its true beliefs to induce DM to take a decision that E 
prefers, thereby causing DM to take a suboptimal decision. For 
instance, in Fig. [2] if E's true beliefs p lie in Ri, its preferred de- 

3 Assuming "ties" at boundaries are broken consistently, the regions 
Di will be convex, but possibly open. 
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Figure 2: Expert utility function. E's utility for each decision 
di is given by the corresponding hyperplane (here, thin black 
line). i5's "optimal" utility B* is the PWLC function shown 
by the dotted green line (i.e., the upper surface), with each Ri 
denoting the regions of belief space where D* (p) = di. Regions 
Di represent DM's policy, where 7r(p) = di for p £ Di. The 
thick red lines denote the (discontinuous) utility B n that E will 
receive from (truthfully) reporting her belief. 

cision is di ; but truthful reporting will induce DM to take decision 
d$. E has greater inherent utility for reporting (any) r £ D\. In- 
deed, its gain from the misreport is p • (bi — hs). Equivalently, E 
stands to lose p • (bi — h^) by reporting truthfully. Intuitively, a 
proper scoring rule would remove this incentive to misreport. 

3.2 Compensation Rules 

If DM knows E utility function, it could reason about E's in- 
centive to misreport and revise its decision policy accordingly. Of 
course, this would naturally lead to a Bayesian game requiring anal- 
ysis of its Bayes-Nash equilibria, and generally leaving DM with 
uncertainty about £"s true beliefs^ Instead, we wish to derive a 
scoring rule that DM can use to incentivize E to report truthfully. 

A compensation function C : A(X ) x X — > R is a mapping 
from reports and outcomes into payoffs, exactly like a standard 
scoring rule. Unlike a scoring rule, however, C does not fully de- 
termine _E's utility for a report; one must also take into account the 
inherent utility E derives from the decision it prompts the DM to 
take. Any compensation function G induces a net scoring function: 



S(p,Xi) = C(p,Xi) + B n (p,Xi 



(5) 



_B's expected net score for report r under beliefs p is S(r, p) = 
C(r, p) + B n (r, p), where C(r, p) = £\ PiC'(r, Xi) is £"s ex- 
pected compensation. 

One natural way to structure the compensation function is to use 
C to compensate E for the loss in inherent utility incurred by re- 
porting its true beliefs p (relative to its best report). This would 
remove any incentive for E to misreport. We define a particular 
compensation function Gi that accounts for this loss: 



Ci(p,Xi) 



^z,7r(p) ■ 



(6) 



Ci (p, Xi) is simply the difference between £"s realized utility for 
its optimal decision (relative to its report p) and the actual decision 
she induced. Gi does not satisfy the usual properties of scoring 
rules: it is given by a subgradient of the loss function, which is 
not convex, nor even continuous. However, E's payoff for a report 
consists of both this compensation and its inherent utility, i.e., her 



4 See Dimitrov and Sami (vj and Conitzer | 6 | for just such a game- 
theoretic treatment of prediction markets (without decisions). 



Si(p, Xi) = Ci(p, Xi) + B 7r (p,x i 



*(p) 



^i,7r(p) 



+ bi^ 



(p)J 



(7) 
(8) 
(9) 



_B's expected net score under beliefs p is identical to her expected 
utility for the optimal decision D* (p). Hence, no other report can 
induce a decision that gives her greater utility. Informally, this 
shows that truthful reporting is optimal. It can be seen directly 
by observing that the net score Si can be derived from Eq.[2]by 
letting G(p) = B*(p) = max 4 < n _Bi(p) be E's optimal utility 
function (which is PWLC, hence convex), and using the subgradi- 
ent G*(p) given by the hyperplane corresponding to the optimal 
decision D* (p) at that point^j 

DEFINITION 2. A compensation function C is proper iff the ex- 
pected net score function S satisfies S(p, p) > S^q, p) for all 
p, q £ A(X). C is strictly proper if the inequality is strict. 

We don't prove this formally since we prove a more general result 
below, but the above informal argument shows: 

PROPOSITION 3. Compensation function C\ is proper. 

REMARK 4. We've defined the compensation function using the 
space of all decisions D. However, this may cause DM to compen- 
sate E for decisions it will never take. If we restrict attention to 
those decisions in the range of DM's policy it, then the above char- 
acterization still applies (and will typically reduce total compen- 
sation). In what follows, we assume the set of decisions has been 
pruned to include only those d £ D for which ir~ (d) 7^ 0, and 
that E's utility function is defined relative to that set. 

Compensation function Gi, while proper, is not strictly proper. 
The induced net scoring function Si is characterized by a non- 
strictly convex cost function G, since G = B* . In particular, for 
any region R(d) of belief space where a single decision d is opti- 
mal for E, every report p £ R(d) has the same expected net score, 
hence there is no "positive" incentive for truthtelling. 

While Ci gives us one mechanism for proper scoring with self- 
interested experts, we can generalize the approach to provide a 
complete characterization of all proper (and strictly proper) com- 
pensation functions. We derived Ci by compensating E for its loss 
due to truthful reporting. This approach is more "generous" than 
necessary. Rather than compensating E for its loss, we need only 
remove the potential gain from misreporting. The key component 
of Ci is not the "compensation term" &;.d*( p ), but rather the the 
penalty term — &i i7r ( p ). It is this penalty that prevents E from ben- 
efiting by changing DM's decision. Any such gain is subtracted 
from its compensation by the inclusion of — &i l7r ( p ). We insist only 
that the positive compensation term is convex: it need bear no con- 
nection to E's actual utility function to incentivize truthfulness^] 
Indeed, we can fully characterize the space of proper and strictly 
proper compensation functions: 

THEOREM 5. A compensation rule C is proper for E iff 

C(p, Xi) = G( P ) - G*(p) • p + G*(p) - b iMp) (10) 

for some convex function G, and subgradient G* of G. C is strictly 
proper iff G is strictly convex. 

5 At interior points of _E's decision regions, the hyperplane is the 
unique subgradient. At _B's decision boundaries, an arbitrary sub- 
gradient can be used. 

incentive to participate is discussed below. 



PROOF. Suppose G is given by Eq. [10] E's utility for a report 
p given outcome x% is given by its net score: 

S(p,Xi) = C(p,Xi) + B n (p,Xi) 

= G(p) - G*(p) ■ p+ G*(p) - 6i^( p ) + 6 ilff (p) 
= G(p)-G*(p)-p + G*(p) 

Since S 1 satisfies the conditions of Thm. [T| the standard proof of 
propriety of S can be used. Similarly, if G is strictly convex, S is 
strictly proper. 

Conversely, suppose G is proper (so that the induced net score 
satisfies 5(p, p) > 5(q,p)). Define G(p) = S(p,p). If 5 is 
proper in this sense, it is easy to show that 5(q, p) is a convex 
function of q (for fixed beliefs p); and since G(p) = S(p, p) = 
maxq 5*(q, p) is the maximum of a set of convex functions (where 
the last equality holds because C is proper), G is itself convex. 
Thm. [T] (or more precisely the method used to prove it) ensures 
that, for some subgradient G* of G, we have S(p, Xi) = G(p) — 
G*(p)-p + G*(p)- Hence, 

C(p,Xi) = S(p,Xi) - B n (p,Xi) 

= G(p)-G*(p)-p + G*(p)-b i><p) 

so G has the required form. If C is strictly proper, then G must be 
strictly convex and Thm.[T]can again be applied. □ 

An illustration of a cost function G(p) that gives rise to a proper 
compensation function is shown in Fig.[3|a). 

The characterization of Thm. [5] ensures truthful reporting, but 
may not provide incentives for participation. Indeed, the expert 
may be forced to pay the DM in expectation for certain beliefs. 
Specifically, if G(p) < B"(p), £"s expected compensation G(p, p) 
is negative. Unless the DM can "force" E to participate, this will 
cause E to avoid providing a forecast if its beliefs are p (e.g., see 
point p is Fig. |3ja)). In general, we'd like to provide E with non- 
negative expected compensation. We can do this by insisting that 
the compensation rule weakly incentives participation: 

DEFINITION 6. A compensation function C satisfies weak par- 
ticipation iff for any beliefs p, E 's expected compensation for truth- 
ful reporting G(p, p) is non-negative. 

(See Fig.[3|b) for an illustration of a cost function G that induces a 
compensation rule G satisfying weak participation.) 

THEOREM 7. A proper compensation rule C satisfies weak par- 
ticipation iff it meets the conditions ofThm.^and G(p) > B T (p) 
for all p £ A(I). 

PROOF. The proof is straightforward: if G(p) > B*(p) for 
all p, then for any truthful report p E's expected compensation is 
G(p) - B n (p) > 0. Conversely, if G(p) < B n {p) for some p, 
then if E holds beliefs p, a truthful report has negative expected 
compensation G(p) — B™(p) < 0. □ 

While weak participation seems desirable, even it is not strong 
enough to ensure an expert's participation in the mechanism in gen- 
eral. Suppose we define a compensation function using some con- 
vex cost function G(p). If E participates, she will maximize her 
net payoff by reporting her true beliefs, say, p. But suppose that 
G(p) < B*(p). While E may not be certain how DM will act 
without its input (e.g., she may not know DM's "default beliefs" 
precisely), she may nevertheless have beliefs about DM's default 
policy. And, if E believes DM will take decision D* (p) if she pro- 
vides no forecast, then she will be better off not participating and 
taking the expected payoff B*(p) derived solely from her inherent 



utility, and forego participation in the mechanism (which limits her 
expected payoff to G(p)). (See point q in Fig.[3|b).) To prevent 
this we can require that G strongly incentivize participation, by in- 
sisting no matter what E believes about DM's default policy (i.e., 
its action given no reporting), it will not sacrifice expected utility 
by participating in the mechanism. 

DEFINITION 8. A compensation function C satisfies strong par- 
ticipation iff, for any decision di £ D, for any beliefs p, E's net 
score for truthful reporting is no less than flj(p). 

Strong participation means that E has no incentive to abstain from 
participation (and need not "take its chances" that DM will make a 
decision it likes). This definition is equivalent to requiring that E's 
expected utility for truthful reporting, as a function of p is at least 
as great as her optimal utility function, i.e., S(p, p) > B*(p) for 
all p £ A(JT). Fig.[5]c) illustrates such a compensation rule. 

THEOREM 9. Proper compensation rule C satisfies strong par- 
ticipation iff it meets the conditions of Thm.^and G(p) > B* (p) 
for all p £ A(I). 

PROOF. The proof is straightforward. Suppose G(p) > B*(p) 
for all p. If E holds beliefs p, then a truthful report has expected 
net score of G(p) > B* (p), and for no beliefs about DM's default 
policy can E derive higher utility by not participating. Conversely, 
suppose G(p) < B* (p) for some p. If E holds beliefs p and also 
believes that DM will take action d — D* (p) if E does not report, 
then E will derive utility B* (p) by not participating, better than 
the optimal expected score G(p) from participating. □ 

OBSERVATION 10. Compensation rule Gi is the unique mini- 
mal (non-strictly) proper rule satisfying strong participation. That 
is, no compensation rule offers lower compensation for any report 
without violating strong participation. 

In general, if we insist on strong participation, DM must provide 
potential compensation up to the level of E's maximum utility gap: 

g(B) = max b ik - b i:j . 

However, this degree of compensation is needed only if DM and 
E have "directly conflicting" interests (i.e., DM takes a decision 
whose realized utility is as far from optimal as possible from £"s 
perspective). In such cases, one would expect E's utility to be sig- 
nificantly less than DM's. If not, this compensation would not be 
worthwhile for DM. Conversely, if E's interests are well aligned 
with those of DM, the total compensation required will be small. 
The most extreme case of well-aligned utility is one where func- 
tions 7r and D* coincide, i.e., 7r(p) = D*(p) for all beliefs p, 
in which case, no compensation is required. Specifically, compen- 
sation function Gi (p) = for all p; and while Ci is not strictly 
proper, the only misreports that E will contemplate (i.e., that do not 
reduce its net score) are those that cannot change DM's decision 
(i.e., cannot impact DM's utility). As a consequence, DM should 
elicit forecasts from an expert who either (a) has well-aligned in- 
terests in the decisions being contemplated; (b) has interest whose 
magnitude is small (hence requires modest compensation) relative 
to DM's own utility; or (c) can be "forced" to make a prediction 
(possibly at negative net cost)Q 

4. POLICY AND UTILITY UNCERTAINTY 

We now relax two key assumptions underlying our compensation 
rule from Section [3T| that E knows DM's policy, and that DM 
knows E's utility function. 

7 For instance, managers may require forecasts from expert employ- 
ees under conditions of negative expected cost. 




Figure 3: Illustration of the cost functions G that correspond to strictly proper compensation rules. In each figure, E's optimal utility 
B* is the PWLC function shown in green, and E's inherent utility B" is the discontinuous function in red. The net scoring function 
G(p) = S(p, p), the convex curve, induces an expected compensation function G(p, p) by subtracting B". (a) A strictly proper rule 
that violates weak participation at point p. (b) A rule that satisfies weak participation but violates strong participation at point q. (c) 
A rule that satisfies strong participation. 



4.1 Policy Uncertainty 

We first consider the case where DM does not want to disclose its 
policy to E. For example, suppose DM wanted to forego a truthful 
compensation rule C and simply rely on a proper scoring rule of 
the usual form that ignores the E's inherent utility. Thm.|5]shows 
that DM cannot prevent misreporting in general if it ignores E's in- 
herent utility; hence it can suffer a loss in its own utility. However, 
by refusing to disclose its policy n, DM could reduce the incentive 
for E to misreport. Without accurate knowledge of n, E would be 
forced to rely on uncertain beliefs about -n to determine the utility 
of a misreport, generally lowering its incentive. However, this will 
not remove the misreporting incentive completely. For instance, re- 
ferring to Fig. [2] suppose DM does not disclose it. HE believes 
with sufficient probability that the decision boundary between c?3 
and di is located at the point indicated, it will misreport any fore- 
cast p in region D3 sufficiently close to that boundary should DM 
use a scoring rule rather than a compensation rule. As such, refus- 
ing to disclose its policy can be used to reduce, but not eliminate, 
the incentive to misreport if DM does not want to use a proper com- 
pensation rule|^] 

Our analysis in the previous section assumed that E used it knowl- 
edge of 7r to determine the report that maximizes her net score. 
However, DM does not need to disclose tt to make good use of a 
compensation rule. It can specify a compensation rule implicitly by 
announcing its net scoring function S(p, Xi) (or the cost function 
G and subgradient G*) and promising to deduct Bd ■ p from this 
score for whatever decision d it ultimately takes. E need not know 
in advance what decision will be taken to be incentivized to offer 
a truthful forecast. Nor does E ever need to know what decisions 
would have been taken had it reported differently. Thus the only 
information E needs to learn about n is the value of 7r(p) at its 
reported forecast p; and even this need not be revealed until after 
the decision is taken (and its outcome realized)]^] 

4.2 Uncertainty in Expert Utility 

We now consider the more interesting issues that arise when DM 
is uncertain about the parameters b of E's utility function. If the 
DM has a distribution over b, one obvious technique is to spec- 
ify a proper compensation rule using the expectation of b. This 
may work reasonably well in practice, depending on the nature of 
the distribution; but it follows immediately from Thm. [5] that this 

8 A similar argument shows that a stochastic policy can be used to 
reduce misreporting incentive, e.g., the soft max policy that sees 
DM take decision di with probability proportional to e Aui< - p '. . 
'Some mechanism to verify the decision post hoc may be needed 
in some circumstances, but this is no different than requiring verifi- 
cation of the realized outcome in standard models of scoring rules. 



approach will not induce truthful reporting in general. 

Rather than analyzing probabilistic beliefs, we instead suppose 
that DM has constraints on b that define a bounded feasible re- 
gion B C R mn in which E's utility parameters must lie. We will 
confine our analysis to a simple, but natural class of constraints, 
specifically, upper and lower bounds on each utility parameter; i.e., 
assume DM has upper and lower bounds bjjf and bijl, respectively, 
on each bij. This induces a hyper-rectangular feasible region B. If 
B is a more general region (e.g., a polytope defined by more general 
linear constraints), our analysis below can be applied to the tightest 
"bounding box" of the feasible region^] Again by Thm.|5]it is clear 
that DM cannot define a proper compensation rule in general: with- 
out certain knowledge of E's utility, any proposed "deduction" of 
inherent utility from E's compensation could mistaken, leading to 
an incentive to misreport. However, this incentive can be bounded. 

Under conditions of utility uncertainty, it is natural for DM to 
restrict its attention to "consistent" compensation rules: 

DEFINITION 11. Let B be the set of feasible expert utility func- 
tions. A compensation rule is consistent with B iff it has the form, 
for some ( strictly ) convex G: 

C{p, Xi ) = G(p) - G*(p) • p + G*(p) - b iMp) (11) 
for some b £ B. 

Notice that consistent compensation rules are naturally linear: in- 
tuitively, we select a single consistent estimate of each parameter 
bij S [bijl, bij[], treat E as if this were her true (linear) utility func- 
tion, and define C using this estimate. Let's say DM is S-certain of 
E's utility iff bif[ — bijl < 8 for all i, j. Then we can bound the 
incentive for E to misreport as follows: 

THEOREM 12. If DM is S-certain ofE 's utility, then E 's incen- 
tive to misreport under any consistent compensation rule is bounded 
by 2S. That is, S(r, p) - 5(p, p) < 2S. 

PROOF. Let p be E's actual beliefs and r some report. 

s ( r , P) = l G ( r ) ~ + b l(r) ] ■ p 

< G(r) • p + S 

< G(p) ■ p + S 

< [G(p) - b w(p) + b w (p) + 5] ■ p + S 
<S(p,p) + 25 

□ 

'"General linear constraints on E's parameters could be could be 
inferred, for example, from observed behavior. 



Notice that the proof assumes that: (a) the estimated utility b w ( r ) 
for the decision induced by E's report r underestimates her true 
utility by 8; and (b) the estimated utility b^p) for the optimal de- 
cision overestimates E's true utility by S. We can limit the mis- 
reporting incentive further by using a uniform compensation rule. 

DEFINITION 13. A consistent compensation rule is uniform if 
each parameter is estimated by 6j,,r(p) = A&ijJ, + (1 — \)bif\ for 
some fixed A £ [0,1]. 

For example, if DM uses the lower bound (or midpoint, or upper 
bound, etc.) of each parameter interval uniformly, we call its com- 
pensation rule uniform. 

COROLLARY 14. If DM is 8-certain of E's utility, then E's in- 
centive to misreport under any uniform compensation rule is bounded 
by 8. That is, S(r, p) - 5(p, p) < 8. 

While bounding the incentive to misreport is somewhat useful, 
it is more important to understand the impact such misreporting 
can have on DM. Fortunately, this too can be bounded. The (strict) 
convexity of G means that the greatest incentive to misreport occurs 
at the decision boundaries of DM's policy tv in Thm.|12| Since, by 
definition, DM is indifferent between the adjacent decisions at any 
decision boundary, misreports in a bounded region around decision 
boundaries have limited impact on DM's utility, as we now show. 
Specifically, we show that the amount by which E will misreport 
is bounded using the "degree of convexity" of the cost function G, 
which in turn bounds how much loss in utility DM will realize. 

DEFINITION 15. Let G be a convex cost function with subgra- 
dient G*. We say G is robust relative to G* with factor m>0iff, 
/oraHp.qe A(X)Q 

G(q) > G(p) + G*(p)-(q-p)+m||q-p||a (12) 

It is not hard to see that m-robustness of the pair G, G imposes 
a minimum "penalty" on any expert misreport, as a function of its 
distance from her true beliefs: 

OBSERVATION 16. Let C be a proper compensation rule based 
on an m-robust cost function G and subgradient G*. Let S be the 
induced net scoring function. Then 

S(p,p) - ^(q.p) > m||q- p|| 2 . 

Together with Thm.|12| this gives a bound on the degree to which 
an expert will misreport when an uncertain DM uses a consistent 
compensation rule. 

COROLLARY 17. Let DM be 8-certain of E's utility and use a 
consistent compensation rule based on an m-robust cost function 
and subgradient. Let p be E's true beliefs. Then the report q 
that maximizes E's net score satisfies ||q — p> 1 1 2 < — • If the 
compensation rule is uniform, then \ | q — p 1 1 2 < Jr. 

In other words, _E's utility-maximizing report must be within a 
bounded distance of her true beliefs if DM uses an m-robust cost 
function to define its compensation rule. 

The notion of m-robustness is a slight variant of the notion of 
strong convexity fij in which we use the specific subgradient G* 
to measure the "degree of convexity." In the specific case of twice 
differentiable cost function G, we say G is strongly convex with 

"The definition of m-robustness can be recast using any reasonable 
metric, e.g., Li-norm or KL-divergence; but the Z/2-norm is most 
convenient below when we relate robustness to strong convexity. 



factor m iff V 2 G(p) y ml for all p£ A(X); i.e., if the matrix 
V 2 G(p) — ml is positive definite 1 1 jj^m-convexity is a sufficient 
condition for the robustness we seek. 

COROLLARY 18. Let DM be 8-certain of E's utility and use a 
consistent compensation rule based on an m-convex, twice differ- 
entiable cost function G. Let p be E 's true beliefs. Then the report 

q that maximizes E's net score satisfies 1 1 q — p 1 1 2 < \ f^- If the 

compensation rule is uniform, then | |q — p| I2 < \f^- 

PROOF. G's assumed differentiability ensures its gradient VG 
is the unique subgradient. Since G is m-convex, we have 

G(q) > G(p) + VG T (p)(q -p) + y||q-p||2 

for all p, q £ A(X) (see 1 1 1). Hence £"s loss in compensation is at 
least y I |q ~~ Pi 1 1- Since its gain in inherent utility by misreporting 
is bounded by 28 (Thm. [T2| (, setting the former to be no greater 
than 28 yields the result. Since the gain in inherent utility under 
a uniform compensation rule is 8, the stronger bound follows by 
substituting 8 for 28 in the preceding argument. □ 

Robustness — and strong convexity if we use a differentiable cost 
function — allow us to globally bound the maximum degree to which 
E will misreport. This allows us to give a simple, global bound on 
the loss in DM utility that results from its uncertainty about the ex- 
pert's utility function. Recall that DM's utility function Ui for any 
decision di is linear, hence has a constant gradient VfJj. (We abuse 
notation and simply write VUi for VUi (p).) The function Ui — Uj 
is also linear, given by parameter vector — Uj). Let denote 
the n-dimensional unit vector with a 1 in component k and zeros 
elsewhere. 

THEOREM 19. Let DM be 8-certain of E's utility and use a 
consistent compensation rule based on an m-robust cost function 
and subgradient. Assume E reports to maximize her net score. 
Then DM's loss in utility relative to a truthful report by E is at 
most maxjj [e^ maxij V((7i — £/•,)] ym" 2 ^. If the compensation 
rule is uniform, then the bound is tightened by a factor of two. 

PROOF. By Cor. [n] E's utility maximizing report q has an 
1/2 distance at most^^ from her true beliefs p. By the Cauchy- 
Schwartz inequality we have j|q — p||i < \A*||q — p] 1 2 , hence 
bounding its max Li-deviation at </n—. Then DM's loss for any 
(utility-maximizing) misreport is: 

U d(p) ■ P — U d(q) ' P 

= u d(p) ■ q + VE/ d (p) (p - q) - u d{q) ■ q - Vf/ d(q) (p - q) 

< V(7 d(p) (p - q) - W d(q) (p - q) 

< V[t/ d(p) -t/ d(q) ](p-q) 

< max S7(Ui - Uj)(p - q) 

28 

< maxfe^ max \7(Ui — Uj)]y/n — . 

k i,j m 

Here the first inequality holds by virtue of u d ( p ) • q < u d ( q ) • q 
(since u d ( q ) is DM's optimal decision at q). □ 

The same proof can be adapted to strongly convex cost functions. 



12 Alternative definitions exist for non-differentiable G, but we as- 
sume a twice differentiable G when discussing strong convexity 
and use robustness relative to a specific subgradient G* for non- 
differentiable G. 
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Figure 4: A locally strongly convex cost function G. Here G has 
is more strongly convex in the neighborhood of decision bound- 
ary D12 than the boundary D23. This means an expert willing 
to sacrifice compensation (e.g., to gain inherent utility due to 
DM uncertainty) can offer a report that deviates more from its 
true beliefs in the neighborhood around D23 and than it can in 
the neighborhood around D12 for the same loss in compensa- 
tion. However, since d-i and d- A are more similar than di and c?2 
(w.r.t. DM utility), i.e., the gradient V((/2 — Us) \ is less than 
I V(f/i — l/a)!, DM will lose less utility "per unit" of misreport 
in the neighborhood of Z>23. Note: the cost function G is drawn 
above DM's utility function U for illustration only — in general, 
it will lie below U. 

COROLLARY 20. Let DM be 5-certain of E's utility and use 
a linear compensation rule based on an m-convex, twice differ- 
entiable cost function G. Assume E reports to maximize her net 
score. Then DM's loss in utility relative to a truthful report by E is 

at most maxfc [ej max^j V(f/j — Uj)]yfnA^. If the compensation 
rule is uniform, then the bound is tightened by a factor of two. 

The results above all rely on the global robustness or global 
strong convexity of the cost function G. Designing a specific cost 
function (and if not differentiable, choosing its subgradients) can 
be challenging if we try to ensure uniform m-robustness or m- 
convexity across the entire probability space A(X). But recall 
that E can only impact DM's utility if its misreport causes DM 
to change its decision. This means that the cost function need only 
induce strong penalties for misreporting near decision boundaries. 
Furthermore, the strength of these penalties should be related to 
the rate at which DM's utility is negatively impacted. For example, 
suppose p lies on the decision boundary between region Di and 
Dj. If |V(f/j — Uj)\ is large, then a misreport in the region around 
p will cause a greater loss in utility than if |V(L7j — Uj)\ is small. 
This suggests that the cost function should be more strongly convex 
(or more robust) near decision boundaries whose corresponding de- 
cisions differ significantly in utility, and can be less strong when the 
decisions are "similar." See Fig.|4]for an illustration of this point. 
Furthermore, the cost function need only be robust or strongly con- 
vex in a local region around these decision boundaries. In particu- 
lar, suppose G is m-robust in some local region around the decision 
boundary between Di and Dj . The degree of robustness bounds the 
maximum deviation from truth that E will contemplate. If the re- 
gion of m-robustness includes these maximal deviations, that will 
be sufficient to bound DM's utility loss for any true beliefs E has in 
that region. Outside of these regions, no misreport by E will cause 
DM to change its decisions (relative to a truthful report). 

We can summarize this as follows: 

DEFINITION 21. G is locally robust relative to G* in the e- 
neighborhood around p with factor m > iff, for all q £ A(X) 



s.t ||q-p||a < e: 

G(q) > G(p) + G*(p)-(q-p) + m||q-p|| a (13) 

Local strong convexity is defined similarly. 

Now suppose DM wishes to bound its loss due to misreporting 
by E by some factor a > 0. This can be accomplished using a 
locally robust cost function: 

THEOREM 22. Let DM be S-certain of E's utility and fix a > 
0. For any pair of decisions di, dj with non-empty decision bound- 
ary Dij, define 

max k (el'V[lJ i -Uf])^n2S a 

Let G be a convex cost function with subgradient G* such that, for 
all i,j and any p £ Dij, (a) G is locally robust with factor rriij 
in the Sij -neighborhood around p; (b) no other decision boundary 
lies within the ey -neighborhood around p. Let DM use a consis- 
tent compensation rule based on G, G*. Assume E reports to max- 
imize her net score. Then DM's loss in utility relative to a truthful 
report by E is at most a. If the compensation rule is uniform, the 
result holds with both rriij and decreased by a factor of two. 

PROOF. (Sketch). The proof proceeds by cases involving the 
location of E's true beliefs p and the location of possible utility- 
maximizing misreports r. W.l.o.g., assume that p is in decision 
region Di. We consider four classes of misreports. 

(A) Suppose r £ Di. In this case, DM's utility loss is zero since 
the decision is the same as if E had reported truthfully. 

(B) Now consider the case where p is in the Sij -neighborhood 
of some decision boundary Dij. We show that any report r £ Dj 
satisfies the condition of the theorem. Let q £ Dj be an arbitrary 
point s.t. I |q — p| I2 < £ij (this must exist by the assumption that p 
lies in the Eij -neighborhood of Dij ). DM's utility loss for reporting 
q is then bounded as follows: 

Ui'p-Uj P 

= Ui q + X7Ui(p — q) — Uj • q+ V(7j(p - q) 

< (p-q) 

< max[efe V(f/< — Z7j)]||p — q||i 

< max[efe V((7i — Uj)]y/n\ |p — q| 1 2 

k 

< max[efc V(f/» — Uj)]y/neij 

k 

= O. 

If r £ Dj, then it must be such a q (i.e., be within Eij of p), since 
any report in Dj has the same inherent utility, while those closest 
to p maximize compensation. Hence utility loss for r is no greater 
than a. Note that p may lie within the neighborhood of multiple 
decision boundaries Dij adjacent to Di, but the argument holds for 
any report in any such region Dj . 

(C) Now consider the case where boundary Dij exists, but p 
does not lie within the -neighborhood of Dij . We show that E's 
utility maximizing report cannot be in Dj. By way of contradiction, 
consider a report r £ Dj. Let £ be the closed line segment {(1 — 
A)p + Ar : A £ [0, 1]}; and let q £ Dij be the point where i 
intersects the decision boundary, and let p' £ Di be the point on 
I on the u Di side" of the boundary that is distance sy from the 
boundary. E's loss in net score (ignoring any error due inherent 



utility misestimate by DM) is given by: 
5(p,p)-S(r,p) 

= Hp ■ p - H r • P 

We have (Z/ p • p — _ff p / • p) > by the propriety of the compensa- 
tion rule (ignoring error due to inherent utility misestimation). We 
also have 

H P ' ■ P - Hq ■ p 

= H P ' • (p - p' + p') - Ha • (p - p' + p') 

= Hp, ■ p - Hq ■ p' + Hp, ■ (p - p') - Hq • (p - p') 

> mijEij + Hp/ ■ (p - p') - Hq ■ (p - p') 

^ TTlij S{j 

> 28 

where the first inequality holds due to the local robustness of G 
and the second due to the convexity of G and the collinearity of 
(p, p', q). Finally, we must have (ff q ■ p — H T • p) > again 
due to the convexity of G and the collinearity of (p, q, r). Thus 
E's loss in compensation due to misreporting is at least 28 (and is 
strictly greater if G is strictly convex). But by Thm.|12|its gain in 
inherent utility by misreporting can be no greater than 28. Hence 
its optimal report r cannot lie in Dj . 

(D) The preceding argument can be adapted in a straightforward 
way to the case where Di and Dj are not adjacent (i.e., Dij is 
empty). □ 

This result can be generalized to the case where the degree of ro- 
bustness around one decision boundary is relaxed sufficiently so 
that the neighborhood within which E can profitably misreport 
crosses more than one decision boundary (i.e., when another deci- 
sion boundary overlaps the Eij -neighborhood around Dij). Utility 
loss will increase but is can be bounded by considering the maxi- 
mum gradient V (Ui — Uj) over decisions that can be swapped. The 
result can also be adapted to locally strongly convex cost functions 
in the obvious way. 

COROLLARY 23. Let DM be 8-certain of E's utility and fix 
a > 0. For any pair of decisions di , dj with non-empty decision 
boundary Dij, define 

_max fc (e£v[t7»-tt,-])Vn25 _ a 
mii ~ a ; £%J "max^efVfC/,-^])^' 

Let G be a convex cost function such that, for all i,j and any 
p £ Dij, (a) G is locally convex with factor rriij in the e%j- 
neighborhood around p; (b) no other decision boundary lies within 
the Eij -neighborhood around p. Let DM use a consistent compen- 
sation rule based on G, G* . Assume E reports to maximize her net 
score. Then DM's loss in utility relative to a truthful report by E is 
at most a. 

These results quantify the "cost" to the decision maker of its impre- 
cise knowledge of the expert's utility function, i.e., its worst-case 
expected utility relative to what it could have achieved if it had full 
knowledge of E's utility (i.e., with truthful reporting by E). 

REMARK 24. If we relax the constraint that DM choose the de- 
cision di with maximum expected utility, we can exploit local ro- 
bustness to induce truthful forecasts. Suppose DM uses the softmax 
decision policy (see footnote^: this stochastic policy makes E's 
utility B 7T (v,\>) continuous in its report r. An analysis similar to 
that above, using local robustness or strong convexity of the cost 



function, allows DM to induce truthtelling as long as the degree of 
convexity compensates for the gradient of at decision bound- 
aries. Since adding randomness to the policy removes the discon- 
tinuities in B w , this is now possible. Of course, this "incentive- 
compatibility" comes at a cost: the DM is committed to taking 
suboptimal actions with some probability. We defer a full analy- 
sis of the tradeoffs, and the relative benefits of "acting optimally " 
but risking misleading reports vs. "acting suboptimally " relative to 
truthful report, to a longer version of this paper. 

The characterization of DM loss using local robustness or lo- 
cal strong convexity not only offers theoretical guarantees on DM 
utility — it has potential operation significance in the design of com- 
pensation rules. Specifically, it suggests an optimization procedure 
for designing a cost function G — from which the induced com- 
pensation rule C is recovered — so as to minimize DM utility loss. 
Intuitively, the design of G will attempt to optimize two conflicting 
objectives: minimizing the bound a on utility loss, which gener- 
ally requires increasing the degree of robustness or convexity of 
G at decision boundaries; and minimizing expected compensation 
c which, given the requirement of strict convexity of G, gener- 
ally requires decreasing robustness or convexity. This tension can 
be addressed by either: (a) explicitly trading a and c off against 
each other in the design objective; (b) minimizing c subject to a 
target bound a\ or (c) minimizing a subject to a target compen- 
sation level c. The optimization itself is defined over the space 
of n-dimensional convex curves G, and could be treated as an n- 
dimensional spline problem. The objective is to fit a convex func- 
tion to a set of points with specific local curvature constraints that 
enforce a certain degree of local convexity at particular decision 
boundaries. Specific classes of spline functions (e.g., Catmull-Rom 
splines) might prove useful for this purpose. We leave to future re- 
search the question of the practical design of cost and compensation 
functions under conditions of utility uncertainty. 

5. MARKET SCORING RULES 

Space precludes a comprehensive treatment, but we provide a 
brief sketch of how one might exploit compensation functions in 
settings where DM aggregates the forecasts of multiple experts. 
One natural means of doing so is to develop a market scoring rule 
( MSR )\9. 1 1 1 that sequentially applies a standard scoring rule based 
on how an expert alters the prior forecast (see Sec.|2j. The typical 
means of creating an MSR given a scoring rule S is to have the fcth 
expert (implicitly) pay the k— 1st expert for its forecast according 
to S, and have the principal pay only final expert for its forecast 
using S. In this way, the principal's total payment is bounded by 
the maximal possible payment to a single expert 

When one attempts this with self-interested experts, difficulties 
emerge. For instance, Shi et al. [15| show that experts who can 
alter the outcome distribution after making a forecast, each require 
compensation to prevent them from manipulating the distribution 
in ways that are detrimental to the principal^] A related form of 
subsidy arises in our decision setting. 

Following [15], we assume a collection of n experts, each of 
whom can provide alter the forecast p exactly once. Suppose the 
experts have an interest in DM's decision. An "obvious" MSR in 
our model would simply adopt a proper compensation rule, and 
have each expert pay the either the compensation or the net score 
due to the expert who provided the incumbent forecast, and receive 
her payment from the next expert. If we use compensation, we run 
into strategic issues. With a proper compensation rule, an expert 

13 Shi et al. [15] actually use a one-round variant of an MSR. 



fc reports truthfully based on her net score (total utility), consisting 
of both compensation and the inherent utility of the decision she 
induces. In a market setting, fc's proposed decision may be changed 
by the next expert that provides a forecast. This (depending on her 
beliefs about other expert opinions) may incentivize k to misreport 
in order to maximize her compensation rather than her net score. 
Overcoming such strategic issues seems challenging. 

Alternatively, each expert might pay the net score due her prede- 
cessor. Unfortunately, an arbitrary proper compensation rule may 
not pay expert k enough score to "cover her costs" (e.g., if k— l's 
inherent utility is much higher than k's). However, if we set aside 
issues associated with incentive for participation for the moment, 
the usual MSR approach can be adapted as follows: we fix a sin- 
gle (strictly) convex cost function G for all experts, and define the 
compensation rule C k for expert k using G in the usual way: 

C k ( P , an) = G(p) - G*(p) ■ P + G*(p) - 6^ (p) , 

where h k is k's utility function (bias). If G satisfies strong partic- 
ipation for all experts (i.e., if G(xi) > B*(xi) for all i), then any 
expert k whose beliefs p[fc] differ from the forecast p[fc— 1] pro- 
vided by fc— 1 will have an expected net score (given p[fc]) greater 
than her expected payment to fc— 1 and will maximize her utility by 
providing a truthful forecast. In particular, let's denote fc's expected 
payment to k—1 by p(k, k—1); then we have: 

p(fc,fc-l) = (# p[ ^i] - b^ [fc _ 1]} ) ■ p[k] + b^p [fc _ 1D • p[k] 

= # P [fc-i] ■ p[k] 
< H P [k] ■ p[k]. 

Hence fc's expected payment p(k, k—1) is less than its expected net 
utility, leaving it with a (positive) net gain of (H p ^ — Hpt^^) • 
p[fc]. However, this gain may be smaller than the inherent utility 
she derives from the decision induced by k—1, namely, b^pj^jj ■ 
p[fc] . Hence this scheme may not incentivize participation. In cases 
where DM can force participation, such a scheme can be used; but 
in general, the self-subsidizing nature of standard MSRs cannot be 
exploited with self-interested experts p] 

To incentivize participation, DM can subsidize these payments. 
In the most extreme case, DM simply pays each displaced expert 
its net utility, which removes any incentives to misreport, but at 
potentially high cost. In certain circumstances, we can reduce the 
DM subsidy to the market by having it pay only the inherent util- 
ity 6^( p [fc_ 1 ]) (given realized outcome xt) of the displaced expert 
k—1, and requiring the displacing expert k to pay the compensa- 
tion -Hi lP [fc-i]- Under certain conditions on the relative utility of 
different experts for different decisions, this is sufficient to induce 
participation; that is, fc's net gain for partipating exceeds her inher- 
ent utility for the incumbent decision. 

For instance, suppose all experts have the same utility function 
b (e.g., consider experts in the same division of a company who 
are asked to predict the outcome of some event, and have different 
estimates, but have aligned interests in other respects). In this case, 



If expert utility is small relative to overall compensation, we can 
exploit the strong robustness (or strong convexity) of the cost func- 
tion to show that experts will abstain from offering predictions only 
if their beliefs are sufficiently close to the incumbent prediction. 
Providing the degree of compensation induced by an "extremely 
convex" cost function can, of course, be interpreted as a form of 
subsidy. 



fc's net gain for reporting her true beliefs is: 

(Hp[k] - (H p [k-i\ - b^( p [ fe _i]))) ■ p[fc] 

= (H p [ k ] - # P [fc-i]) ■ p[fc] + b T (p [fe _i]) • p[fc] 
> b^p^i]) ■ p[fc]. 

Hence fc's expected net gain is at least as great as her inherent ex- 
pected utility for the decision induced by k—1, and strictly greater if 
her beliefs differ from those of k—1. Thus participation is assured. 

Indeed, the argument holds even if the utility functions are not 
identical: we require only that fc's expected utility for the decision 
it displaces is less than the expected utility (given fc's beliefs) to be 
offered to her predecessor k—1. A sufficient condition for this is 
that h k < b fc_1 (pointwise). This suggests that if the DM can elicit 
predictions of its experts in a particular order, it should do so by 
eliciting forecasts of those with the greatest utility first. 

In general, even in the extreme case of identical expert utility 
functions, there seems to be no escape from the requirement that 
DM subsidize the market, at a level that grows linearly with the 
number of agents. This is very similar to the conclusions drawn 
by Shi et al. |15| . We provide a more detailed formalization and 
analysis in an extended version of the paper. 

6. CONCLUDING REMARKS 

We have presented a model that allows the analysis of the incen- 
tives facing experts in a decision-making context who have a vested 
interest in the decision taken by the principal. We have developed 
a class of compensation rules that are necessary and sufficient to 
induce truthful forecasts from self-interested experts and also char- 
acterized the subclasses of such rules that satisfy weak and strong 
participation constraints. While vanilla compensation rules assume 
knowledge of the expert's utility function on the part of the prin- 
cipal, we've also shown how to design compensation rules when 
the principal has only rough bounds on expert utility parameters 
in such a way that (a) the incentive for the expert to misreport is 
bounded; and (b) the impact on the principal's decision/utility is 
similarly bounded relative to the case of full knowledge. These 
bounds are derived from the robustness or strong convexity of the 
cost function, in either a global or a local sense. 

A number of other interesting directions remain. One is the de- 
velopment of computationally effective procedures to design cost 
functions that minimize principal utility loss in settings where ex- 
pert utility is not fully known, without inducing extreme degrees 
of compensation. Another direction is the analysis of the tradeoff 
between the principal's decision space and the payments required 
to induce truthfulness or participation on the part of experts. We 
discussed above the possibility that by acting (somewhat) subop- 
timally through the use of a stochastic policy, the principal could 
diminish the incentive for the expert to misreport. We can take this 
a step further: intuitively, by limiting its policy to use only a subset 
of its potential decisions, the principal may dramatically reduce the 
potential for strategic behavior — either misreporting or failure to 
participate — on the part of an expert, while at the same time, doing 
little damage its own utility by restricting its policy in this wayQ 

Finally, the question of joint elicitation both the utility function 
and the forecast of an expert remains intriguing. This can be viewed 
as a mechanism design problem where the expert's type consists of 
both its preferences and its "information" or forecast (see |6| for 
a treatment of forecasts themselves from a mechanism design per- 
spective). Preliminary results (joint with Tuomas Sandholm) sug- 

15 Thanks to Tuomas Sandholm for suggesting this and preliminary 
discussion in this direction. 



gest, not surprisingly, that truthful elicitation is not generally possi- 
ble when the principal takes a decision that maximizes its expected 
utility. However, it remains to be seen if effective mechanisms can 
be designed that offer "reasonable" performance from the perspec- 
tive of the principal. 
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